docs: page descriptions, titles, structured data and crawler files for search and answer engines - #651

A crawl of every page in cmcp.agentrust-io.com's sitemap on 2026-09-14 found what search and answer engines were reading:

• 44 of 48 pages carried the same meta description, site_description, because Material falls back to it when a page has none. The quickstart's own description ran to 252 characters.
• Titles repeated the site name: cMCP: policy-checked MCP tool calls with signed evidence - cMCP on the home page, and two pages were both titled Response inspection - cMCP.
• JSON-LD had no Organization @id and repeated an inline Organization as author on every page, 96 objects in all, with nothing describing the page it sat on.
• /robots.txt served only Cloudflare's content-signals comment block, with no directives and no Sitemap line. The repository-root robots.txt never deployed, because docs_dir is docs.
• llms.txt had Getting started, Specification and Tutorials, with no guides beyond four tutorials and no limitations page.

Changes

• hooks/seo.py, registered under hooks: in mkdocs.yml, sets each page's description from its first paragraph of prose, capped at 155 characters. Front matter still wins. The same hook runs on every AgenTrust docs site.
• Four spec pages get a hand-written description: in front matter, because their first paragraph is a label list or a context sentence rather than a summary: failure-modes, phase2-server, tool-identity and transport. The quickstart's is cut to 138 characters.
• overrides/main.html overrides htmltitle: the home page uses its front-matter title alone, and other pages drop the - cMCP suffix when the title already names cMCP. og:title and twitter:title now equal <title>, and og:site_name is AgenTrust.
• The same template emits one JSON-LD graph per page: the Organization under https://agentrust-io.com/#organization, the id the hub and the other docs sites use; a WebSite; a WebPage on the home page or a TechArticle on inner pages, with the page's own title, description and canonical URL; a BreadcrumbList on inner pages; and on the home page the SoftwareApplication node with codeRepository and the MIT license.
• robots.txt moves to docs/robots.txt, names the same ten crawlers as agentrust-io.com, disallows /cdn-cgi/, and points at https://cmcp.agentrust-io.com/sitemap.xml.
• The tutorial nav label becomes "Response inspection tutorial", so the two pages no longer share a title.
• The llms.txt sections follow the tabs: Get started, Specification (eleven pages), Guides (tutorials, deployment, hardware validation), Project (limitations, sponsors), and the description ends by linking agentrust-io.com.
• docs.yml and docs-check.yml also run when hooks/** changes.

Checked

• mkdocs build --strict with requirements/docs.txt exits 0.
• All 48 pages: no two share a description, none is over 160 characters or contains an em or en dash, and no title repeats the site name. Home title: cMCP: policy-checked MCP tool calls with signed evidence.
• Every page has exactly one Organization node with the hub @id, no inline Organization objects, and one page node whose url equals the canonical, whose description equals the meta description, and whose headline equals the title. All 47 inner pages carry a BreadcrumbList.
• robots.txt is at the site root with the Sitemap line, and llms.txt has the four sections.
• python scripts/check_docs_quickstart.py against the checkout: PASS.
• hooks/seo.py imports cleanly under python -W error.

Generated with Claude Code

https://claude.ai/code/session_013aK3gVWzNdcM3hZ2o2awK2